Regression Discontinuity Design

class: center, middle, inverse, title-slide

# Regression Discontinuity Design
## Tutorial 6
### Stanislav Avdeev

---

# Regression discontinuity

- Regression discontinuity design (RDD) is currently the darling of the econometric world for estimating causal effects without running an experiment

The basic idea is this:

- We look for a treatment that is assigned on the basis of being above/below a *cutoff value* of a continuous variable, for example:
- if a candidate gets 50.1% of the vote they're in, 40.9% and they're out
- if you're 65 years old you get Medicaid, if you're 64.99 years old you don't
- if you score above 75, you'll be admitted into a "gifted and talented" (GATE) program

We call these continuous variables "running variables" because we *run along them* until we hit the cutoff

---

# Regression discontinuity

- First, let's simulate the dataset

```r
set.seed(1000)
rdd.data <- tibble(test = runif(1000)*100) %>%
  mutate(GATE = test >= 75) %>%
  mutate(earn = runif(1000)*40+10*GATE+test/2)
```

---

# Regression discontinuity

- Notice that the y-axis here is *In GATE*, not the outcome

![](data:image/png;base64,#tutorial6_files/figure-html/unnamed-chunk-2-1.png)

---

# Regression discontinuity

- Here's how it looks when we consider the actual outcome

![](data:image/png;base64,#tutorial6_files/figure-html/unnamed-chunk-3-1.png)

---

# Regression discontinuity

- If we look at the relationship between treatment and going to college, we'll be picking up the fact that higher test scores make you more likely to go to college anyway

![](data:image/png;base64,#tutorial6_files/figure-html/unnamed-chunk-4-1.png)

---

# Regression discontinuity

- Except, that's not actually what the diagram looks like. Test only affects GATE to the extent that it makes you be above the 75 cutoff

![](data:image/png;base64,#tutorial6_files/figure-html/unnamed-chunk-5-1.png)

---

# Regression discontinuity

- Basically, the idea is that *right around the cutoff*, treatment is randomly assigned
- If you have a test score of 74.9 (not high enough for gifted-and-talented), you're basically the same as someone who has a test score of 75.0 (just barely high enough)
- So we have two groups - the just-barely-missed-outs and the just-barely-made-its, that are basically exactly the same except that one happened to get treatment
- A perfect description of what we're looking for in a control group
- So if we just focus around the cutoff, it's basically random which side of the line you're on
- But we get variation in treatment
- This specifically gives us the effect of treatment *for people who are right around the cutoff* - "local average treatment effect" (we still won't know the effect of being put in gifted-and-talented for someone who gets a 30)

---

# Regression discontinuity

- A very basic idea of this, before we even get to regression, is to create a *binned chart* 
- And see how the bin values jump at the cutoff
- A binned chart chops the Y-axis up into bins
- Then takes the average Y value within that bin
- Then, we look at how those X bins relate to the Y binned values
- If it looks like a pretty normal, continuous relationship, then JUMPS UP at the cutoff X-axis value, that tells us that the treatment itself must be doing something

---

# Simulation

The true effect is `$10$`

```r
#Choose a "bandwidth" of how wide around the cutoff to look (arbitrary in our example)
#Bandwidth of 2 with a cutoff of 75 means we look from 75-2 to 75+2
bandwidth <- 2
#Just look within the bandwidth
rdd <- rdd.data %>% filter(abs(75-test) < bandwidth) %>%
  #Create a variable indicating we're above the cutoff
  mutate(above = test >= 75) %>%
  #And compare our outcome just below the cutoff to just above
  group_by(above) %>% summarize(earn = mean(earn))
rdd
```

```
## # A tibble: 2 × 2
##   above  earn
##   <lgl> <dbl>
## 1 FALSE  55.2
## 2 TRUE   66.0
```

```r
#Our effect looks just about right
rdd$earn[2] - rdd$earn[1]
```

```
## [1] 10.80055
```

---

# Graphically

![](data:image/png;base64,#tutorial6_files/figure-html/unnamed-chunk-7-1.gif)

---

# Regression discontinuity in regression

- First, we need to *transform our data*
- We need a "Treated" variable that's `TRUE` when treatment is applied - above or below the cutoff
- Then, we are going to want a bunch of things to change at the cutoff. This will be easier if the running variable is *centered around the cutoff*. So we'll turn our running variable `$X$` into `$X - cutoff$` and call that `$XCentered$`

Let's start with the simple linear version:

$$ Y = \beta_0 + \beta_1XCentered + \beta_2Treated + \beta_3Treated\times XCentered +\varepsilon $$
- `$\beta_2$` is how the intercept jumps - that's the RDD effect
- `$\beta_3$` is how the slope changes - that's the RKD effect
- Sometimes the effect of interest is the interaction term - the change in slope. This answers the question "does the effect of `$X$` on `$Y$` change at the cutoff? This is called a "regression kink" design

---

# Regression discontinuity in regression

The true effect is `$0.7$`

```r
set.seed(2000)
df <- tibble(X = runif(1000)) %>%
  mutate(treated = X > .5) %>%
  mutate(X_centered = X - .5) %>%
  mutate(Y = X_centered + .7*treated + .5*X_centered*treated + rnorm(1000,0,.3))
lm(Y ~ treated*X_centered, data = df)
```

```
## 
## Call:
## lm(formula = Y ~ treated * X_centered, data = df)
## 
## Coefficients:
##            (Intercept)             treatedTRUE              X_centered  
##               -0.01113                 0.74669                 0.98250  
## treatedTRUE:X_centered  
##                0.44696
```

You can take this basic interaction-with-cutoff design idea and use it to look at how *anything* changes before and after cutoff, not just the level of `$Y$`. You could look at how the *slope* changes ("regression kink")

---

# Graphically

- The true model is an RDD effect of `$0.7$`, with a slope of `$1$` to the left of the cutoff and a slope of `$1.5$` to the right

![](data:image/png;base64,#tutorial6_files/figure-html/unnamed-chunk-9-1.png)

---

# Choices

- Bandwidth choice
- Functional form 
- Controls

---

# Bandwidth choice

- The idea of RDD is that people *just around the cutoff* are very much comparable
- Basically random if your test score is 74 vs. 76 if the cutoff is 75, for example
- So people far away from the cutoff aren't too informative. At best they help determine the slope of the fitted lines
- So we might limit our analysis within just a narrow window around the cutoff
- This makes the exogenous-at-the-jump assumption more plausible, and lets us worry less about functional form (over a narrow range, not too much difference between a linear term and a square), but on the flip side reduces our sample size considerably
- Imbens and Gelman (2018) show that the "naive" RDD estimators place high weights on observations far from the threshold
- So it's better to drop these observations

---

# Bandwidth choice

- RDD generally uses data only from the observations in a given range around the cutoff
- Or at least weights them less the further away they are from cutoff
- How wide should the bandwidth be?
- There's a big wide literature on *optimal bandwidth selection* which balances the addition of bias (from adding people far away from the cutoff who may have back doors) vs. variance (from adding more people so as to improve estimator precision)
- We won't be doing this by hand, we can often rely on an RDD command to do this for us
- The `rdrobust` package in R implements some state of the art literature on this subject.

---

# Bandwidth choice

- Pay attention to the sample sizes, accuracy (true value `$0.7$`) and standard errors

```r
m1 <- lm(Y~treated*X_centered, data = df)
m2 <- lm(Y~treated*X_centered, data = df %>% filter(abs(X_centered) < .25))
m3 <- lm(Y~treated*X_centered, data = df %>% filter(abs(X_centered) < .1))
m4 <- lm(Y~treated*X_centered, data = df %>% filter(abs(X_centered) < .05))
m5 <- lm(Y~treated*X_centered, data = df %>% filter(abs(X_centered) < .01))
export_summs(m1,m2,m3,m4,m5, statistics = c(N = 'nobs'), coefs = 'treatedTRUE')
```

<table class="huxtable" style="border-collapse: collapse; border: 0px; margin-bottom: 2em; margin-top: 2em; ; margin-left: auto; margin-right: auto;  " id="tab:unnamed-chunk-10">
<col><col><col><col><col><col><tr>
<th style="vertical-align: top; text-align: center; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;"></th><th style="vertical-align: top; text-align: center; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0.4pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">Model 1</th><th style="vertical-align: top; text-align: center; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0.4pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">Model 2</th><th style="vertical-align: top; text-align: center; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0.4pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">Model 3</th><th style="vertical-align: top; text-align: center; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0.4pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">Model 4</th><th style="vertical-align: top; text-align: center; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0.4pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">Model 5</th></tr>
<tr>
<th style="vertical-align: top; text-align: left; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">treatedTRUE</th><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">0.75 ***</td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">0.77 ***</td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">0.71 ***</td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">0.61 ***</td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">0.56 </td></tr>
<tr>
<th style="vertical-align: top; text-align: left; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;"></th><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0pt 0pt 0.4pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(0.04)   </td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0pt 0pt 0.4pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(0.06)   </td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0pt 0pt 0.4pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(0.09)   </td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0pt 0pt 0.4pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(0.15)   </td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0pt 0pt 0.4pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(0.43)</td></tr>
<tr>
<th style="vertical-align: top; text-align: left; white-space: normal; border-style: solid solid solid solid; border-width: 0pt 0pt 0.8pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">N</th><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0.8pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">1000       </td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0.8pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">492       </td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0.8pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">206       </td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0.8pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">93       </td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0.8pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">15    </td></tr>
<tr>
<th colspan="6" style="vertical-align: top; text-align: left; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;"> *** p < 0.001;  ** p < 0.01;  * p < 0.05.</th></tr>
</table>

---

# Functional form

- Why fit a straight line on either side? If the true relationship is curvy this will give us the wrong result
- We can be much more flexible. As long as we fit some sort of line on either side, we can look for the jump
- The way to do this is with polynomials

$$Y = \beta_0 + \beta_1XCentered + \beta_2XCentered^2 + \beta_3Treated + $$

`$$\beta_4Treated\times XCentered + \beta_5Treated\times XCenrtered^2 + \varepsilon$$`

- `$\beta_3$` remains our "jump at the cutoff" - our RDD estimate

---

# Functional form

The true effect is `$0.7$`

```
## 
## Call:
## lm(formula = Y ~ X_centered * treated + I(X_centered^2) * treated, 
##     data = df)
## 
## Coefficients:
##                 (Intercept)                   X_centered  
##                    -0.03397                      0.69904  
##                 treatedTRUE              I(X_centered^2)  
##                     0.76774                     -0.57215  
##      X_centered:treatedTRUE  treatedTRUE:I(X_centered^2)  
##                     0.75094                      0.53190
```

---

# Functional form

- The interpretation is the same as before - look for the jump
- We want to be careful with polynomials though, and not add too many
- Remember, the more polynomial terms we add, the stranger the behavior of the line at *either end* of the range of data
- And the cutoff is at the far-right end of the pre-cutoff data and the far-left end of the post-cutoff data
- So we can get illusory effects generated by having too many terms
- A common approach is to use *non-parametric* regression or *local linear regression*
- This doesn't impose any particular shape. And it's easy to get a prediction on either side of the cutoff
- This allows for non-straight lines without dealing with the issues polynomials bring us

---

# Functional form

- Looking purely just at the cutoff and making no use of the space *away* from the cutoff throws out a lot of useful information
- We know that the running variable is related to outcome, so we can probably improve our *prediction* of what the value on either side of the cutoff should be if we *use data away from the cutoff to help with prediction* than if we *just use data near the cutoff*, which is what that animation does
- We can do this with OLS
- The bin plot we did can help us pick a functional form for the slope

---

# Functional form

- Let's look at the same data with a few different functional forms
- Remember, the RDD effect is the jump at the cutoff. The TRUE effect here will be `$0.7$`, and the TRUE model is an order-2 polynomial

```r
tb <- tibble(Running = runif(200)) %>%
  mutate(Y = 1.5*Running - .6*Running^2 + .7*(Running > .5) + rnorm(200, 0, .25)) %>%
  mutate(X_centered = Running - .5, Treated = Running > .5)
```

---

# Functional form

![](data:image/png;base64,#tutorial6_files/figure-html/unnamed-chunk-14-1.png)

---

# Functional form

![](data:image/png;base64,#tutorial6_files/figure-html/unnamed-chunk-15-1.png)

---

# Functional form

![](data:image/png;base64,#tutorial6_files/figure-html/unnamed-chunk-16-1.png)

---

# Functional form

![](data:image/png;base64,#tutorial6_files/figure-html/unnamed-chunk-17-1.png)

---

# Functional form

![](data:image/png;base64,#tutorial6_files/figure-html/unnamed-chunk-18-1.png)

---

# Functional form

![](data:image/png;base64,#tutorial6_files/figure-html/unnamed-chunk-19-1.png)

---

# Functional form

- Avoid higher-order polynomials
- Even the "true model" can be worse than something simpler sometimes
- And fewer terms makes more sense too once we apply a bandwidth and zoom in
- Be very suspicious if your fit veers wildly off right around the cutoff
- Consider a nonparametric approach

---

# Controls

- Generally you don't need control variables in an RDD
- If the design is valid, you've closed all back doors. That's sort of the whole point
- Although maybe we want some if we have a wide bandwidth - this will remove some of the bias
- Still, we can get real value from having access to control variables. How?
- Control variables allow us to perform *placebo tests* of our RDD model
- We can rerun our RDD model, but simply use a control variable as the outcome
- We should not find any effect (outside of the levels expected by normal sampling variation)
- You can run these for *every control variable you have*

---

# Balance

- One thing that's so great about RDD is that, since it's basically random whether you're on one side of the cutoff or another, there shouldn't be other back doors
- It's a form of within variation that's *so narrow* it basically closes everything
- We can check this by seeing if other variables differ on either side of the line

---

# Assumptions

- We knew there must be some assumptions lurking around here
- Some are more obvious (we should be using the correct functional form)
- Others are trickier. What are we assuming about the error term and endogeneity here?
- Specifically, we are assuming that *the only thing jumping at the cutoff is treatment*
- Sort of like parallel trends, but maybe more believable since we've narrowed in so far
- For example, if having an income below 150% of the poverty line gets you access to food stamps AND to job training, then we can't really use that cutoff to get the effect of just food stamps
- The only thing different about just above/just below should be treatment
- What if the running variable is *manipulated*?

---

# Manipulated running variables

- Imagine you're a teacher grading the gifted-and-talented exam. You see someone with an 74 and think "aww, they're so close! I'll just give them an extra point..."
- Suddenly, that treatment is a lot less randomly assigned around the cutoff
- If there's manipulation of the running variable around the cutoff, we can often see it in the presence of *lumping*
- In other words, there's a big cluster of observations to one side of the cutoff and a seeming gap missing on the other side
- How can we check this?
- We can look graphically by just checking for a jump at the cutoff in *number of observations* after binning

---

# Manipulated running variables

- Here's an example from the real world in medical research - statistically, p-values *should* be uniformly distributed
- But it's hard to get insignificant results published in some journals. So people might "p-hack" until they find some form of analysis that's significant, and also we have heavy selection into publication based on `$p < .05$`. Can't use that cutoff for an RDD

![p-value graph from Perneger & Combescure, 2017](data:image/png;base64,#p_value_distribution.png)

---

# Manipulated running variables

- The first one looks pretty good. We have one that looks not-so-good on the right

![](data:image/png;base64,#tutorial6_files/figure-html/unnamed-chunk-20-1.png)

---

# Manipulated running variables

- Another thing we can do is do a "placebo test"
- Check if variables *other than treatment or outcome* vary at the cutoff
- We can do this by re-running our RDD but just swapping out some other variable for our outcome
- If we get a significant jump, that's bad. That tells us that *other things are changing at the cutoff* which implies some sort of manipulation (or just super lousy luck)

---